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ABSTRACT 

The research aims to examine whether there is a difference in undergraduate student performance on ski 11-based exams 
in an introductory computer literacy course at a state comprehensive university when exams are administered in-class 
vs. online. Two samples, each consisting of approximately 107 students, are considered for this study. A comparison of 
exam scores will be used to identify differences in exam performance between the two groups. 


INTRODUCTION 

At increasing rates, institutions of higher education are 
using technology in the presentation of course content. 
The availability and popularity of online learning is 
growing. The number of institutions offering Web-based 
courses are increasing. In the United States, 56% of 
colleges and universities are offering online courses (Liu, 
2005). As universities move towards the development of 
online/distance courses, they are faced with a number of 
challenges. One of these challenges is how to 
adequately assess student learning in an online 
environment. 

The transition of a traditional course to an online or hybrid 
course requires faculty to evaluate their assessment 
strategies. As with assessment in traditional courses, the 
development of an assessment portfolio within a course 
should be linked to the school's mission, degree specific 
learning goals, and course specific learning goals. 
Instructors need to match appropriate assessment 
techniques (e.g., homework, case projects, exams, 
presentations, participation, etc.) with the specific 
objectives of each course. There is also some attention 
given in the literature on effective techniques to assess 
student learning in an online environment (Bryant, S.M., 
Kahle, J.B., and Schafer, B.A., 2005; Gaytan, 2005; Robles 
and Braathen, 2002). 

The most commonly reported challenge in online 
assessment is how to maintain academic integrity (which 
is a challenge in traditional courses as well); specifically, 
how do we catch cheating, verify the identity of the 
student, and curtail plagiarism (Byrd and Lott, 2003; 


Scanlon, 2003). Scanlon (2003) further asserts that no 
existing technology can ensure academic honesty. 
Commonly the literature "encourages faculty to hold 
examinations on campus thereby ensuring a higher 
degree of academic honesty" (Gaytan 2005; Alexander, 
Truell, and Bartlett, 2002). The assumption is that online 
testing will increase academic dishonesty and therefore 
result in inflated student performance. 

Most of the works discussing performance in online 
courses focus on comparing student learning in an online 
course with a traditional course (Anstine, J. and Skidmore, 
M., 2005; Liu, Y., 2006). Other works focus on the impact 
of learning styles in students' preference for various 
pedagogical techniques used in distance or hybrid 
based courses (Becker, K., Kehoe, J., and Tennent, B., 
2007). 

Clearly, faculty are concerned with maintaining 
academic integrity in their classes. Overcoming the 
concern that students will commit more acts of 
academic dishonesty in online/distances courses is 
critical to obtain. This research offers insight into 
quantifying the impact of varying exam environment on 
student performance and on students 
attitudes/perceptions towards the course. 

Hybrid Course versus Traditional Course Description 
The course under discussion is INF0273, Introduction to 
the Computer in Business, an essential computer literacy 
course allotted for all undergraduate business majors 
within the syllabi. This course focuses on teaching 
computer literacy in the Microsoft Office products. Word, 
PowerPoint, Excel, and Access 2003. Assessment in the 


1-manager’s Journal of Educational Technology, Vol. 4 • No. 2 • July - September 2007 


63 






RESEARCH PAPERS 


course has three main components - homework, case 
studies, and examinations. The focus in course 
assessments is twofold to assess students' competency 
with basic computer literacy as well as their ability to apply 
computer skills to solve "real-world" problems in business 
environment. 

Prior to describing the hybrid course model studied, it is 
useful to describe the traditional Computers in Business 
Course. The traditional course is delivered in an 
instructional computer lab and is a hands-on course. 
Course sections are limited to 30 students; each student is 
assigned a computer in the lab. Instructors teach 
students through a combination of computer 
demonstrations, in-class exercises, hands-on homework 
assignments and case studies. Exams are administered 
through online testing software, SAM2003, a product from 
Course Technology. 

Course content does not differ from the traditional to the 
hybrid course. Course sections in the hybrid course are 
limited to 120 students; students in each section are 
assigned to two class meeting times: (1) a common 
session (1.5 hrs/week) where all 120 students meet in a 
large lecture - room and (2) a breakout session (1.5 
hrs/week) where groups of 30 students meet in a hands-on 
computer lab. The common session is used for four 
mandatory lectures throughout the semester that provide 
an introduction to various topics within the course. The 
breakout session is used for the instructor and graduate 
assistants to provide one-on-one assistance to students 
completing coursework. In the hybrid course, the primary 
method of instruction is through the use of online course 
tutorials in SAM2003. The material in SAM2003 is divided 
into modules that correspond to chapters in the course 
textbooks. A suggested pace for completing the online 
training modules is provided to students through the 
course syllabus, and online through the course 
management system. 

Milestones in the form of homework assignments are 
provided to assist students to keep pace in the course; 
students submit homework assignments through SAM 
Prograder, a product from Course Technology. Student 
homework assignments are automatically graded by 


computer through the SAM Prograder program; based 
upon student performance reports, students can revise 
and resubmit homework assignments through the system 
multiple times to improve their grade and understanding 
of course material. In addition to homework assignments, 
students are required to complete a hands-on exams 
using SAM2003 and a case study where students apply 
their computer skills to solve "real-world" business 
problems for each section of the course. 

Research Scope and Hypothesis 
This paper, examines whether there is a difference in 
undergraduate student performance on skills-based 
exams in an introductory computer literacy course at a 
state, comprehensive university when exams are 
administered in-class vs. online. The study also 
challenged the hypothesis that student exam 
performance will be different in the group, who take their 
exams online. The hypothesis examined in this research is 
as follows: 

Hypothesis 1 :Students taking their exams online will 
achieve the same level of performance as students 
taking their exams in-class. 

In addition, student perceptions of the hybrid computer 
course are gathered through participant observations, 
course evaluation surveys, and informal interviews. 

Data Collection 

Subjects for this research are, students enrolled in two 
sections of an introductory computer literacy course at a 
state university. Both sections of the course are taught by 
the same professor and when choosing their section of 
the course, students were not aware that there were any 
differences in assessment methods in the two sections. 
Each section of the course had enrolled with 
approximately 107 students. Sections meet at similar 
times during the day. Additionally, both sections are 
identical in form, curriculum, and types of assessment 
(i.e., homework, projects, and exams). The only difference 
in the two sections is that Section 1 took all three exams in 
a proctored computer lab while section 2 took their three 
exams without monitoring. Students in both sections are 
required to take each exam during the same one week 
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period of time. 

Data for this project was gathered through: 

• Recording student perceptions of the hybrid 
computer course through participant observations, 
course evaluation surveys, and informal interviews 

• Analyzing student performance by section through 
three sets of graded hands-on exams. 

Hands-on Exams 

Topics and Testing Environment 

All exams in the course are administered in a hands-on 
computer environment. Exams are skills-based and 
delivered using a publisher developed Computer Based 
Test (CBT); during the period of study SAM2003 was used, 
which is developed by Thompson Course Technology. 
Three exams are administered during the course of the 
semester; one for each module in the course: 
Word/PowerPoint, Excel, and Access. Each hands-on 
exam is comprised of 30-40 questions; students are 
allotted a 60 minute window of time to complete each 
exam. All questions require students to complete a skills- 
based task in Word, PowerPoint, Excel, or Access. 
Example questions include: 

• Insert a footnote at the end of the third paragraph with 
the text "The name of the course is INF02 73". 

• Apply the "Concourse" design theme to all slides in 
the presentation. 

• Insert a formula in cell B10 that calculates the product 
of cells B8 and B9. 

• Create a select query that returns to the title and 
publisher of all books written by authors with the last 
name Mitchell 

While taking each exam students are allowed two 
attempts to answer each question correctly. If a student 
performs the required skill correctly the first time they are 
automatically advanced to the next question; if a student 
performs an incorrect action, the question is reset and the 
student can take their second attempt. Upon answering 
each question, students know the outcome whether it is 
correct or incorrect. Throughout the exam, students have 
the option of skipping questions and attempting them 


later in the exam. The CBT allows students to perform tasks 
in multiple ways (i.e., using shortcut keys, using menus, or 
using the mouse). A sample screen shot of a skills-based 
question can be viewed in Figure 1. 

Students are automatically graded by the CBT. At the 
completion of the exam (when either all questions are 
answered or the allotted time has expired), students are 
provided a detailed performance report with both their 
overall score and question specific details, including a 
log of all keystrokes made for each question. 

Data Analysis- Student Performance 
To test our research hypothesis, a one-way ANOVA was 
performed on results from the online section and the in- 
class section for each of the three exams as well as the 
average of each student's three exam scores. Table 1 
presents the results ofeachofthefourANOVAs. 


Results, based on each of the three exams as well as the 
average of three exams indicate that the hypothesis 
cannot be rejected at the a=0.05 level of significance. 
As seen in the results presented in Table 1, the p values 
range from .1720 to .8349, all indicating that null 
hypothesis could not be rejected. The ANOVA results 
show that there is no significant difference in student 


Exam 

P-Value 

Word 

.6840 

Excel 

.8349 

Access 

.1720 

Exam Average 

.4050 


Table 1. ANOVA Results 



Figure 1. A skill-based exam question from SAM2003 
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Figure 2. Histogram of Exam Averages for Students 
in the INF0273 Course 


exam performance between the section that took the 
exams in-class in a proctored environment or the section 
that took the exams in an un-proctored setting. 

To examine the academic performance of students in 
the computer applications course, the average of each 
student's average exam score (based on three exams) 
was calculated. Exams evaluated the proficiency in 
Microsoft Word, PowerPoint, Excel, and Access. All exams 
were scored on a one-hundred point scale. The grade 
distribution and analysis is presented in Figure 2. 

As Figure 2 displays, overall students performed well in the 
course. Sixty-five percent of students' average exam 
scores are above 80. Ten percent of students have exam 
averages in the failing range of below 50. 

Data Analysis - Student Perceptions 
Specially designed course evaluation surveys were used 
to gather information, concerning students' experiences 
in the hybrid computer course. 154 of the 214 students 
enrolled in the course have completed the online course 
evaluation survey. 

All data collected was prepared for analysis with Excel 
2003 and SPSS14. Five statements were administered to 
examine student attitudes and perceptions toward the 
hybrid computer class. The summaries that follow are 
based upon entire student responses. Students were 
given a statement and asked if they 'strongly agree, 
agree, uncertain, disagree 7 , or strongly disagree with the 
given statement. It is noted that none of the students 
submitted a 'strongly disagree 7 response to any of the 
survey questions. 

The first statement evaluated students' opinions regarding 


the professors' accessibility. 

Statement 1: The professor is easily accessible and 
encourages students to seek his/her help outside of class: 
Students evaluated the accessibility of their hybrid course 
professor. As shown in Figure 3, the majority of students 
(79.7%) agreed or strongly agreed with the statement that 
their professor was easily accessible and encouraged 
students to seek his/her help outside of class despite the 
fact that there were only four required class meetings. 

The second and third statements evaluated the online 
course content system including the tutorials and the 
practice tests. 

Statement 2: Regarding the SAM 2003 Online Tutorials - 
they were useful in learning Microsoft Office Skills for the 
course: Students overwhelmingly found the online course 
content system to be effective. As shown in Figure 4, a 
total of seventy-eight percent (78%) of the students 
strongly agreed or agreed with the statement. 

Statement 3: The practice tests in SAM 2003; the online 
content system; were helpful in preparing for in-class 
exams: The majority of the students felt that evaluating 



The Professor is easily accessible and encourages 
students to seek his/her help outside of class 



Figure 4. Student Responses to Statement 2: 
Regarding the SAM 2003 Online Tutorials - they were 
useful in learning Microsoft Office Skills for the course 
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Figure 5. Student Responses to Statement 3: 

The practice tests in SAM 2003; the online content system; 
were helpful in preparing for in-class exams. 


their progress with practice tests was helpful for studying 
for in-class exams. As shown in Figure 5, seventy-four 
percent (74%) of the students strongly agreed or agreed 
with the statement. Only six percent (6%) disagreed with 
the statement. 

The fourth statement evaluated the students' opinions on 
the appropriateness of the hybrid format for the 
computers in business course. 

Statement 4: Regarding the Hybrid format of the course - 
the self-paced format of the course was appropriate for 
this course: Based on the response to this statement 
students appeared to have a positive experience with the 
hybrid format of the course. As shown in Figure 6, ninety- 
five percent (95%) of the students strongly agreed or 
agreed that they would consider taking another course 
utilizing the online course content. Only five percent (5%) 
disagreed with the statement. 

The last statement evaluated students' inclination to take 
another hybrid course. 

Statement 5: I would consider another course that is 
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Figure 6. Student Responses to Statement 4: 
Regarding the Hybrid format of the course - the self-paced 
format of the course was appropriate for this course. 
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Figure 7. Student Responses to Statement 5: 

I would consider another course that is 
offered in a hybrid course format 

offered in a hybrid course format: The majority of the 
students reported that they would consider taking another 
hybrid course. As shown in Figure 7, eighty-five percent 
(85%) of the students agreed with the statement. Only five 
percent (5%) disagreed with the statement. 

Conclusion 

Assessment in online courses is a large issue. Many 
instructors are reluctant to teach in this environment 
because they are concerned about who is actually doing 
the work. The assumption is that online assessment will 
result in cheating. 

The results clearly indicate that in the exams 
administered as part of the introduction to computers 
course, there is no difference in student performance 
when exams are administered in a proctored vs. un- 
proctored environment. It can be assumed with the results 
that on these particular exams cheating did not result in a 
disparity in performance between the two sections. 

A possible contributing factor to a non-detectable 
amount of cheating could be the format of the exams 
administered in this course. All course exams are 
delivered through a hands-on simulated office testing 
environment. All exam questions require students to 
perform tasks in a simulated Microsoft Office 
environment. The type of test is not conducive to 
encouraging cheating behavior. 

As a follow-up, the impact of assessment form has on 
cheating behavior of students is also planned to be 
examined. 
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